Computer vision based activation

ABSTRACT

An augmented-reality (AR) device detects a physical device located within a first predefined distance of the augmented-reality device. The AR device detects, using an optical sensor, a physical object located within a second predefined distance of the physical device, the physical object being electronically unconnected to the physical device. The AR device determines that the physical object is associated with the physical device and identifies a command based on an identification of both the physical object and the physical device. The physical device is configured to operate the command and display, in the display, virtual content as an overlay to the physical object.

PRIORITY APPLICATION

This application is a continuation of, and claims priority to U.S.patent application Ser. No. 15/659,180, filed Jul. 25, 2017, thedisclosure of which is incorporated herein in its entirety by reference.

TECHNICAL FIELD

The subject matter disclosed herein generally relates to the processingof data. Specifically, the present disclosure addresses systems andmethods for remotely controlling devices.

BACKGROUND

A device can be used to generate and display data in addition to animage captured with the device. For example, augmented reality (AR) is alive, direct or indirect, view of a physical, real-world environmentwhose elements are augmented by computer-generated sensory input such assound, video, graphics or GPS data. With the help of advanced ARtechnology (e.g. adding computer vision and object recognition) theinformation about the surrounding real world of the user becomesinteractive. Device-generated (e.g., artificial) information about theenvironment and its objects can be overlaid on the real world.

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings.

FIG. 1 is a block diagram illustrating an example of a network suitablefor operating a computer vision based activation of a physical device,according to some example embodiments.

FIG. 2 is a block diagram illustrating modules (e.g., components) of anaugmented reality device, according to some example embodiments.

FIG. 3 is a block diagram illustrating modules (e.g., components) of aphysical device & control application, according to some exampleembodiments.

FIG. 4 is a block diagram illustrating interactions between an augmentedreality device, a server, and a physical device, according to someexample embodiments.

FIG. 5 is a block diagram illustrating interactions between an augmentedreality device, a server, and a physical device, according to someexample embodiments.

FIG. 6 is a block diagram illustrating interactions between an augmentedreality device and a physical device, according to some exampleembodiments.

FIG. 7 is a block diagram illustrating interactions between an augmentedreality device and a physical device, according to some exampleembodiments.

FIG. 8 is a flowchart illustrating an example operation of a physicaldevice control application, according to some example embodiments.

FIG. 9 is a flowchart illustrating another example operation of aphysical device control application, according to some exampleembodiments.

FIG. 10 is a flowchart illustrating another example operation of aphysical device control application, according to some exampleembodiments.

FIG. 11 is a flowchart illustrating another example operation of aphysical device control application, according to some exampleembodiments.

FIG. 12A is a block diagram illustrating an example of an operation ofan augmented reality device for controlling a television.

FIG. 12B is a block diagram illustrating an example of an operation ofan augmented reality device for controlling a television.

FIG. 13A is a block diagram illustrating an example of an operation ofan augmented reality device for controlling a computer.

FIG. 13B is a block diagram illustrating another example of an operationof an augmented reality device for controlling a computer.

FIG. 14A is a block diagram illustrating an example of an operation ofan augmented reality device for controlling a radio.

FIG. 14B is a block diagram illustrating another example of an operationof an augmented reality device for controlling a radio.

FIG. 14C is a block diagram illustrating an example of an augmentedreality display for controlling a radio.

FIG. 15 is a block diagram illustrating components of a machine,according to some example embodiments, able to read instructions from amachine-readable medium and perform any one or more of the methodologiesdiscussed herein.

DETAILED DESCRIPTION

Example methods and systems are directed to remotely controlling adevice by detecting an associated physical object being present inproximity (or being present in a known physical configuration such asproximity to, alignment with, or in a relative position to) to thedevice. Examples merely typify possible variations. Unless explicitlystated otherwise, components and functions are optional and may becombined or subdivided, and operations may vary in sequence or becombined or subdivided. In the following description, for purposes ofexplanation, numerous specific details are set forth to provide athorough understanding of example embodiments. It will be evident to oneskilled in the art, however, that the present subject matter may bepracticed without these specific details.

Augmented reality (AR) applications allow a user to experienceinformation, such as in the form of a three-dimensional virtual objectoverlaid on a picture of a physical object captured by a camera of adevice (also referred to as an augmented-reality device). The physicalobject may include a visual reference that the AR application canidentify. For example, the AR device uses computer vision to detect andidentify the physical object. A visualization of the augmentedinformation, such as the three-dimensional virtual object overlaid orengaged with an image of the physical object is generated in a displayof the device. The three-dimensional virtual object may be selectedbased on the recognized visual reference or captured image of thephysical object. A rendering of the visualization of thethree-dimensional virtual object may be based on a position of thedisplay relative to the visual reference.

In one example, the AR device may include a head wearable device thatincludes a transparent display. The AR device displays the augmentedinformation in the transparent display such that the user of the ARdevice perceives the augmented information as an overlay on a physicalobject.

The present application describes using an AR device that activates andcontrols a physical device (e.g., computer, smart phone, telephone,radio) by detecting that a physical object (e.g., keys, business card,kitchen utensils) associated with the physical device is in proximity tothe physical device. In one example embodiment, the physical object isnot electrically coupled to the physical device and does not communicatewith the physical device or another other device. In another exampleembodiment, the physical object includes electrical components (e.g., atelevision remote). For example, the television remote is notcommunicating or electrically coupled to the television. Therefore, atelevision remote without batteries can still be used to activate andcontrol the television.

For example, the AR device includes a camera that detects and identifiesa laptop. The AR device also detects that a predefined physical object(e.g., coffee mug) is placed to the right side of the laptop. The ARdevice detects the coffee mug using different methods such as usingcomputer vision for object recognition or using other wireless means(e.g., bluetooth chip embedded in the coffee mug) to identify thepresence of the coffee mug. Once the AR device determines that thecoffee mug is associated with the laptop, the AR device displaysaugmented information (e.g., displaying a color filter over the coffeemug to turn it orange) and turn on the laptop. In another example, theAR device determines that the coffee mug (associated with the laptop) isplaced to the left side of the laptop. The AR device displays augmentedinformation (e.g., “sleeping five seconds” notification, or a virtualcount down timer) over or above the laptop or the coffee mug asperceived by the user of the AR device.

In another example, the AR device triggers a command to turn off thetelevision (e.g., AR device controls a smart wi-fi switch connected tothe television) when the AR device detects that the television remotecontrol is placed under the television. Conversely, the AR devicetriggers a command to turn on the television when the AR device detectsthat the television remote control is removed from a predefined location(e.g., next to the television). In yet another example, the AR devicetriggers a command (e.g., to the television, or a set top box connectedto the television) to set the television to a first channel (e.g.,channel A) when the AR device detects that the television remote controlis positioned face up. Similarly, the AR device triggers a command toset the television to a second channel (e.g., channel B) when the ARdevice detects that the television remote control is positioned facedown. The AR device displays augmented information such as “switching tochannel B” over the television remote control.

In one example embodiment, the AR device includes an optical sensor anda display. The AR device detects a physical device located within afirst predefined distance of the augmented-reality device. The AR devicedetects, using the optical sensor, a physical object located within asecond predefined distance of the physical device, the physical objectbeing electronically unconnected to the physical device. The AR devicedetermines that the physical object is associated with the physicaldevice, identifies a command based on an identification of both thephysical object and the physical device (the physical device isconfigured to operate the command), and displays, in the display,virtual content as an overlay to the physical object, the virtualcontent identifying the command.

In another example embodiment, the AR device determines, using theoptical sensor, a position of the physical object relative to thephysical device, a first orientation of the physical object relative tothe augmented-reality device, and a second orientation of the physicalobject relative to the physical device. The command is identified basedon a combination of the identification of the physical object, theposition of the physical object relative to the physical device, thefirst orientation of the physical object relative to theaugmented-reality device, and the second orientation of the physicalobject relative to the physical device.

In another example embodiment, the AR device identifies, using theoptical sensor, the physical device and the physical object. The ARdevice determines, using the optical sensor, that the physical object isadjacent to the physical device. The command is based on theidentification of both the physical device and the physical object, andthe physical object being adjacent to the physical device.

In another example embodiment, the AR device wirelessly communicateswith the physical device, identifies the physical device based on thewireless communication between the augmented reality device and thephysical device, determines, using the optical sensor, that the physicalobject is adjacent to the physical device. The command is based on theidentification of both the physical device and the physical object, andthe physical object being adjacent to the physical device.

In another example embodiment, the AR device communicates the command tothe physical device using the wireless communication. The AR device cancommunicate the command to a server (the server being associated withthe physical device and coupled to the physical device).

In another example embodiment, the AR device detects that the physicaldevice has performed the command on the physical device, and modifiesthe virtual content in response to the detecting that the physicaldevice has performed the command. The modified virtual content indicatesthat the physical device has performed the command.

In another example embodiment, the AR device receives a confirmationfrom the physical device that the command has been performed on thephysical device. The AR device modifies the virtual content in responseto detecting that the physical device has performed the command. Themodified virtual content indicates that the physical device hasperformed the command.

In another example embodiment, the AR device detects, using the opticalsensor, a first and a second physical object located within thepredefined distance of the physical device, the first and secondphysical objects being electronically unconnected to the physicaldevice. The AR device determines that the first and second physicalobjects are associated with the physical device, identifies a firstrequest based on a combination of an identification of the first andsecond physical objects, the position of the first physical objectrelative to the physical device, the position of the second physicalobject relative to the physical device, the orientation of the firstphysical object relative to the augmented-reality device, and theorientation of the of the second physical object relative to theaugmented-reality device. The AR device then communicates the firstrequest to the physical device (or to an electrical device such as asmart plug or a set top box connected to the physical device).

In another example embodiment, the AR device generates a first virtualcontent to be displayed over at least one of the first and secondphysical objects, the first virtual content identifying the firstrequest. The AR device generates a second virtual content identifying asecond request, the second virtual content including visual instructionsillustrating how to move the first and second physical objects totrigger the second request. The AR device displays, in the display, boththe first and second virtual content over the first and second physicalobjects.

In another example embodiment, a non-transitory machine-readable storagedevice may store a set of instructions that, when executed by at leastone processor, causes the at least one processor to perform the methodoperations discussed within the present disclosure.

FIG. 1 is a block diagram illustrating an example of a networkenvironment 100 suitable for operating an AR device, according to someexample embodiments. The network environment 100 includes an AR device104, a physical device 106, and a server 110 communicatively coupled toeach other via a computer network (e.g., the Internet 112).

The physical device 106 includes an electrical device that can beactivated and controlled remotely. Examples of the physical device 106include but are not limited to a computer, a smart phone, a machine, atelevision, a radio, and a vehicle. In another example embodiment, thephysical device 106 may be an electrical or non-electrical device thatis not capable of electrical communication with the AR device 104. Forexample, a conventional lamp is an example of an electrical devicehaving no capability of electrically communicating with other devicessuch as the AR device 104. A door is an example of a non-electricaldevice having no capability of electrically communicating with otherdevices such as the AR device 104. However, the conventional lamp ordoor are coupled to an electrical device that communicates with the ARdevice 104. For example, the conventional lamp is connected to a smartswitch (e.g., wifi or zwave switch). The door is connected to a wirelesscontrollable mechanical actuator. The AR device 104 can thereforecontrol electrical and non-electrical physical devices through anotherelectrical device (e.g., wi-fi switch, wireless controllable mechanicalactuator) capable of controlling the electrical and non-electricaldevices and capable of wireless communication with the AR device 104.

The AR device 104 detects the physical device 106 and the physicalobject 108 and controls the physical device 106 based on the detectionof the physical device 106 and physical object 108. The AR device 104determines that the physical object 108 is associated with the physicaldevice 106 and is located in proximity to the physical device 106. Inone example embodiment, the AR device 104 accesses a database ofpredefined relationships between physical devices 106 and physicalobjects 108. For example, computer c1 is associated with a mug with logoor design d1, computer c2 is associated with a mug with logo or designd2, machine m1 is associated with tool (e.g., a wrench) t1, machine m2is associated with tool t2. The database further defines predefinedactions or commands between the physical devices 106 and the physicalobjects 108 and their relative location or placement. For example,command a1 is associated with the mug with design d1 being placed to theright of the computer c1. Command a2 is associated with the mug withdesign d2 being placed to the right of the computer c1. Command a3 isassociated with the mug with design d1 being placed. An action orcommand may also be associated with the identification of both thephysical device 106 and the physical object 108 irrespective of theirrelative placement. For example, the AR device 104 turns on a televisionwhen the AR device 104 detects both the television and the remotecontrol, irrespective of the locaiton of the remote control. The ARdevice 104 detects the presence of both the television and the remotecontrol using a combination of wireless radio signal and computervision.

The AR device 104 displays augmented reality information overlaid on topof the physical device 106 or a physical object 108. For example, theuser 102 sees a virtual logo on a television network displayed on top ofa television when the AR device 104 detects that a remote control isplaced face up on a stand next to the user 102.

In one example embodiment, the AR device 104 includes a wearablecomputing device (e.g., smart glasses, a smart visor, smart eyewear, asmart helmet, a smart phone) that a user 102 can wear to see orexperience the AR content related to physical device 106 and thephysical object 108, or other predefined physical objects at a specificgeographic location or position in a real-world physical environment.FIG. 1 illustrates an example embodiment of the user 102 looking at orthrough a display of the AR device 104 to view an AR experience (e.g.,augmented information) as an overlay to the physical device 106 and thephysical object 108.

The user 102 may be a user of an AR application executed in the ARdevice 104 or the server 110. The user 102 may be a human user (e.g., ahuman being), a machine user (e.g., a computer configured by a softwareprogram to interact with the AR device 104), or any suitable combinationthereof (e.g., a human assisted by a machine or a machine supervised bya human). The user 102 is not part of the network environment 100, butis associated with the AR device 104.

In another example embodiment, the AR device 104 retrieves andcommunicates a command (e.g., turn on, turn off, change to channel B)directly to the physical device 106 or to an electrical device (e.g., aset top box, a Wi-Fi switch) connected to the physical device 106. Inanother example embodiment, the AR device 104 communicates the commandto the server 110 that is coupled to the physical device 106. Forexample, the AR device 104 determines that the physical device 106 ismade by manufacturer M. The AR device 104 identifies a server 110associated with the manufacturer M and provides the command to theidentified server 110.

The AR device 104, the physical device 106, and the server 110 may eachbe implemented in a computer system, in whole or in part, as describedbelow with respect to FIG. 15.

The server 110 may be part of a network-based system. For example, thenetwork-based system may be or include a cloud-based server system thatprovides AR content (e.g., AR experience including three-dimensional(3D) models of virtual objects, animations, images, and video) to the ARdevice 104.

Any of the machines, databases, or devices shown in FIG. 1 may beimplemented in a general-purpose computer modified (e.g., configured orprogrammed) by software to be a special-purpose computer to perform oneor more of the functions described herein for that machine, database, ordevice. For example, a computer system able to implement any one or moreof the methodologies described herein is discussed below with respect toFIG. 15. As used herein, a “database” is a data storage resource and maystore data structured as a text file, a table, a spreadsheet, arelational database (e.g., an object-relational database), a triplestore, a hierarchical data store, or any suitable combination thereof.Moreover, any two or more of the machines, databases, or devicesillustrated in FIG. 1 may be combined into a single machine, database,or device, and the functions described herein for any single machine,database, or device may be subdivided among multiple machines,databases, or devices.

The computer network (e.g., the Internet 112) may be any network thatenables communication between or among machines (e.g., physical device106, server 110), databases, and devices (e.g., AR device 104).Accordingly, the computer network may be a wired network, a wirelessnetwork (e.g., a mobile or cellular network), or any suitablecombination thereof. The computer network may include one or moreportions that constitute a private network, a public network (e.g., theInternet 112), or any suitable combination thereof.

FIG. 2 is a block diagram illustrating an example embodiment of the ARdevice 104. The AR device 104 may be a wearable device that includessensors 202, a display 204, a storage device 206, and a processor 208.The AR device 104 may include any type of device that can be worn on thehead of a user (e.g., the user 102) such as glasses, a headband, a hat,or a visor.

The sensors 202 may be used to generate internal tracking data (e.g.,using gyroscope data, accelerometer data, or inertial motion unit data)of the AR device 104 to determine a position and an orientation of theAR device 104. The position and the orientation of the AR device 104 maybe used to identify real-world objects in a field of view of the ARdevice 104. For example, a virtual object may be rendered and displayedin the display 204 when the sensors 202 indicate that the AR device 104detects or is oriented towards a predefined real-world object (e.g.,when the user 102 looks at the physical device 106 using the AR device104). In another example, the AR device 104 displays a virtual object(also referred to as augmented information, virtual content, syntheticcontent) based on a current geographic location of the AR device 104 andthe physical object 108. For example, a specific virtual content andcommand are respectively displayed and issued at the user 102's homeaddress. On the other hand, another virtual content and command arerespectively displayed and issued when the user 102 is located at his orher work address. Therefore, different virtual content and command canbe configured based on the geographic location of the AR device 104, thephysical device 106, and the physical object 108, the identification ofthe physical device 106, and the physical object 108, the relativeposition and location between the AR device 104, the physical device106, and the physical object 108.

Examples of sensors 202 include a camera, an audio sensor, an inertialmotion unit (IMU) sensor, a geographic location sensor, a barometer, ahumidity sensor, an ambient light sensor, and a biometric sensor. It isto be noted that the sensors 202 described herein are for illustrationpurposes. The sensors 202 are thus not limited to the ones described.

The display 204 includes a display surface or lens capable of displayingAR content (e.g., an image of a virtual display or monitor) generated bythe processor 208. The display 204 may be transparent so that the user102 can see through the display 204 (e.g., such as a head-up display).

The storage device 206 stores a library of physical device identifiers,physical object identifiers, AR content, commands, geographic locations,relative locations and positions, a profile of the user 102. The ARcontent include two- or three-dimensional models of virtual objects withor without corresponding audio. In another example, the storage device206 may also store a database that identifies reference objects (visualreferences or unique identifiers associated with the physical device106, physical object 108) and corresponding AR content (e.g.,animations, 3D virtual objects, or interactive features of the 3Dvirtual objects).

In one example embodiment, the AR device 104 communicates over theInternet 112 with the server 110 to access a database of informationrelated to the physical device 106 and physical object 108 from theserver 110, commands related to the identified physical device 106 andphysical object 108 and their relative placement, location, andposition.

The processor 208 includes an AR application 210 and an activation &control application 212. The AR application 210 generates a display ofvirtual content when the AR device 104 detects the physical device 106and the physical object 108 associated with the physical device 106 isin proximity to the physical device 106.

The application & control application 212 detects the presence of thephysical device 106 (e.g., via computer vision, visual objectrecognition, or wireless means such as Bluetooth or Zwave) and thepresence of the physical object 108 associated with the physical device106. After detecting the presence of the physical device 106, and thatthe associated physical object 108 is in proximity to the physicaldevice 106, the physical device control application 212 communicates acommand to the physical device 106 or the server 110 to activate oroperate the physical device 106.

FIG. 3 is a block diagram illustrating modules (e.g., components) of theapplication & control application 212, according to some exampleembodiments. The application & control application 212 includes aphysical device detection module 302, a physical object detection module304, a validation module 306, an activation module 308, and a controlmodule 310. The physical device detection module 302 detects thepresence of the physical device 106 (e.g., using computer vision,wireless communication signal such as Wi-Fi or Zwave). For example, thephysical device detection module 302 detects and identifies the physicaldevice 106. In addition, the physical device detection module 302determines a distance between the AR device 104 and the physical device106, a geographic location of the physical device 106, a relativeposition between the physical device 106 and the AR device 104, anorientation of the physical device 106 relative to the AR device 104(e.g., television facing away from the AR device 104).

The physical object detection module 304 detects the presence of thephysical object 108 (e.g., using computer vision or wirelesscommunication signal such as Bluetooth or Zwave). The physical objectdetection module 304 detects and identifies the physical object 108. Inaddition, the physical object detection module 304 determines a distancebetween the AR device 104 and the physical object 108, a geographiclocation of the physical object 108, a relative position between thephysical object 108, the physical device 106, and the AR device 104(e.g., the physical object 108 is to the right of the physical device106 and in front of the AR device 104), an orientation of the physicalobject 108 relative to the physical device 106 and/or the AR device 104(e.g., remote control is face up and aimed at the television).

The validation module 306 receives information related to theidentification, location, and position of the physical device 106 andthe physical object 108 respectively from the physical device detectionmodule 302 and the physical object detection module 304. The validationmodule 306 retrieves AR content and a command (e.g., turn on, turn off,start application A, switch channel) associated with the receivedinformation. For example, the validation module 306 retrieves a commandand a particular AR content associated with the physical object 108determined to be present within a predefined distance (e.g., 5 feet)from the physical device 106. In another example, the validation module306 retrieves another command and another AR content associated with thephysical object 108 determined to be positioned in a particularorientation relative to the physical device 106 or the AR device 104.

The server module 308 uses the information received from the physicaldevice detection module 302 to identify a server 110 associated with thephysical device 106. For example, the server module 308 accesses aserver 110 associated with the physical device 106. In another example,the server module 308 accesses a server 110 associated with the ARdevice 104 to identify another server 110 associated with the physicaldevice 106. The server module 308 receives the command from thevalidation module 306 and communicates the command to the identifiedserver 110.

The physical device interface module 310 receives the command from thevalidation module 306 and communicates the command (e.g., via theinternet 112 or via a direct wireless communication means such as radio)to the physical device 106. For example, the control module 310 sends acommand or operation to turn off a radio (physical device 106).

FIG. 4 is a block diagram illustrating interactions between the ARdevice 104, the server 110, and the physical device 106, according tosome example embodiments. At operation 402, the AR device 104 detectsthe physical device 106 using computer vision or other wirelesscommunication signal between the physical device 106 and the AR device104. At operation 404, the AR device 104 detects the physical object 108using computer vision recognition techniques or other wirelesscommunication signal between the physical object 108 and the AR device104 (e.g., Bluetooth tag on the physical object 108). At operation 406,the AR device 104 determines that the physical object 108 is associatedwith the physical device 106. The AR device 104 retrieves a command(e.g., activate physical device 106) based on the associated physicalobject 108 and sends a request or instructions to activate the physicalobject 108 to the server 110 (associated with the physical device 106)at operation 408. At operation 410, the server 110 sends the command toactivate the physical device 106 to the physical device 106. Atoperation 412, the physical device 106 is activated in response toreceiving the command.

FIG. 5 is a block diagram illustrating interactions between the ARdevice 104, the server 110, and the physical device 106, according tosome example embodiments. At operation 502, the AR device 104 identifiesa position and orientation of the physical object 108 relative to thephysical device 106 (e.g., one foot to the left of the physical device106, and pointed in a particular direction). At operation 504, the ARdevice 104 determines that a control or command based on the relativelocation and position between the physical object 108 and the physicaldevice 106. At operation 506, AR device 104 sends the command to theserver 110 (associated with the physical device 106). At operation 508,the server 110 sends the command to the physical device 106. Atoperation 510, the physical device 106 performs the command in responseto receiving the command from the server 110.

FIG. 6 is a block diagram illustrating interactions between the ARdevice 104 and the physical device 106, according to some exampleembodiments. At operation 602, the AR device 104 detects the presence ofthe physical device 106 using computer vision or other wirelesscommunication signal between the physical device 106 and the AR device104. At operation 604, the AR device 104 detects the physical object 108using computer vision recognition techniques or other wirelesscommunication signal between the physical object 108 and the AR device104 (e.g., Bluetooth tag on the physical object 108). At operation 606,the AR device 104 determines that the physical object 108 is associatedwith the physical device 106. The AR device 104 retrieves a command(e.g., activate physical device 106) based on the associated physicalobject 108 and sends a request or instructions to activate the physicalobject 108 to the physical device 106 at operation 608. At operation610, the physical device 106 is activated in response to receiving thecommand from the AR device 104.

FIG. 7 is a block diagram illustrating interactions between an augmentedreality device 104 and a physical device 106, according to some exampleembodiments. At operation 702, the AR device 104 identifies a positionand orientation of the physical object 108 relative to the physicaldevice 106 (e.g., one foot to the left of the physical device 106, andpointed in a particular direction). At operation 704, the AR device 104determines that a control or command based on the relative location andposition between the physical object 108 and the physical device 106. Atoperation 706, the AR device 104 sends the command to the physicaldevice 106. At operation 708, the physical device 106 performs thecommand in response to receiving the command from the AR device 104.

FIG. 8 is a flowchart illustrating an example operation of the physicaldevice control application 212, according to some example embodiments.At operation 802, the physical activation & control application 212detects the presence of the physical device 106 and identifies thephysical device 106. In one example embodiment, operation 802 isimplemented with the physical device detection module 302. At operation804, the physical device control application 212 identifies the physicalobject 108 and detects that the physical object 108 is in proximity(e.g., adjacent or 5 feet away) to the physical device 106. In oneexample embodiment, operation 804 is implemented with the physicalobject detection module 304. At operation 806, the physical devicecontrol application 212 determines that the physical object 108 isassociated with a command. In one example embodiment, operation 806 isimplemented with the validation module 306. At operation 808, thephysical device control application 212 sends the command to thephysical device 106. In one example embodiment, operation 808 isimplemented with the physical device interface module 310. At operation810, the AR device 104 displays AR content based on the command. In oneexample embodiment, operation 810 is implemented with the AR application210. For example, the AR content includes virtual content identifying orillustrating a command based on the position or placement of thephysical object 108 relative to the physical device 106. In anotherexample, the AR content includes illustration on how to manipulate thephysical object 108 to perform another command.

FIG. 9 is a flowchart illustrating another example operation of theapplication & control application 212, according to some exampleembodiments. At operation 902, the physical device control application212 determines the position and orientation of the physical object 108relative to the physical device 106 (and optionally relative to the ARdevice 104). In one example embodiment, operation 902 is implementedwith the physical device detection module 302 and the physical objectdetection module 304. At operation 904, the application & controlapplication 212 determines a command based on the relative position andorientation of the physical object 108. In one example embodiment,operation 904 is implemented with the validation module 306. Atoperation 906, the physical device control application 212 communicatesthe command to the physical device 106. In one example embodiment,operation 906 is implemented with the physical device interface module310. At operation 908, the AR device 104 displays AR content based onthe command. In one example embodiment, operation 908 is implementedwith the AR application 210.

FIG. 10 is a flowchart illustrating another example operation of theapplication & control application 212, according to some exampleembodiments. At operation 1002, the application & control application212 detects the physical device 106 using computer vision techniques. Inone example embodiment, operation 1002 is implemented with the physicaldevice detection module 302. At operation 1004, the application &control application 212 detects that the physical object 108 (associatedwith physical device 106) is adjacent to the physical device 106, usingcomputer vision techniques. In one example embodiment, operation 1002 isimplemented with the physical object detection module 304. At operation1006, the application & control application 212 detects that thephysical object 108 is associated with the physical device 106. In oneexample embodiment, operation 1006 is implemented with the validationmodule 306. At operation 1008, the application & control application 212determines a command based on detecting that the physical object 108 isadjacent to the physical device 106. In one example embodiment,operation 1008 is implemented with the validation module 306.

FIG. 11 is a flowchart illustrating another example operation of anactivation and control application, according to some exampleembodiments. At operation 1102, the application & control application212 detects the physical device 106 using wireless communication betweenthe AR device 104 and the physical device 106. In one exampleembodiment, operation 1102 is implemented with the physical devicedetection module 302. At operation 1104, the application & controlapplication 212 detects the physical object 108 (associated withphysical device 106), using computer vision techniques. In one exampleembodiment, operation 1104 is implemented with the physical objectdetection module 304. At operation 1106, the application & controlapplication 212 determines the position and orientation of the physicalobject 108 relative to the AR device 104. In one example embodiment,operation 1106 is implemented with the physical device detection module302 and the physical object detection module 304. At operation 1108, thephysical device control application 212 determines a command based onthe position and orientation of the physical object 108. In one exampleembodiment, operation 1108 is implemented with the validation module306. At operation 1110, the application & control application 212communicates the command to the physical device 106. In one exampleembodiment, operation 1110 is implemented with the physical deviceinterface module 310.

FIGS. 12A, 12B are block diagrams illustrating examples of operation ofan augmented reality device for controlling a television. The user 102uses the AR device 104 to detect a television 1204 and a remote control1206 associated with the television 1204. In one example embodiment, theAR device 104 visually detects the television 1204 and the remotecontrol 1206 both located within a field of view 1202 of an opticalsensor of the AR device 104. The AR device 104 identifies that theremote control 1206 is associated with the television 1204. The ARdevice 104 further determines that the remote control 1206 is adjacentor in proximity (or within a predefined distance) to the television1204. The AR device 104 further detects the orientation of the remotecontrol 1206 relative to the television 1204. FIGS. 12A and 12B bothillustrate the remote control 1206 having different orientation. The ARdevice 104 determines a command based on the position and orientation ofthe remote control 1206 and sends the command to the television 1204. Inthe example of FIG. 12A, the AR device 104 sends a “turn off” signal tothe television 1204 based on the orientation of the remote control 1206.In the example of FIG. 12B, the AR device 104 sends a “turn on” signalto the television 1204 based on the orientation of the remote control1206. It is noted that the user 102 did not activate or press anybuttons on the remote control 1206 to turn on or off the television1204.

FIGS. 13A and 13B are block diagrams illustrating examples of anoperation of an augmented reality device 104 for controlling a computer.The user 102 uses the AR device 104 to detect and identify a computerdisplay 1302 and a pen 1306 associated with the computer display 1302.The computer display 1302 displays an application A 1304 running on acomputer (not shown) connected to the computer display 1302. In oneexample embodiment, the AR device 104 visually detects that the computerdisplay 1302 and the pen 1306 are both located within a field of view1202 of an optical sensor of the AR device 104. The AR device 104identifies that the pen 1306 is associated with the computer or computerdisplay 1302. The AR device 104 further determines that the pen 1306 isadjacent (or within a predefined distance) and placed to the left of thecomputer display 1302. FIGS. 13A and 13B both illustrate the pen 1306 indifferent positions (e.g., to the left, to the right) relative to thecomputer display 1302. The AR device 104 determines a command based onthe location of the pen 1306 relative to the computer display 1302 andsends the command to the computer connected to the computer display1302. In the example of FIG. 13A, the AR device 104 sends a signal torun application A 1304 on the computer connected to the computer display1302 based on detecting that the pen 1306 is located to the left of thecomputer display 1302. In the example of FIG. 13B, the AR device 104sends a signal to run application B 1308 in the computer connected tothe computer display 1302 based on detecting that the pen 1306 islocated to the right of the computer display 1302.

FIGS. 14A and 14B are block diagrams illustrating an example of anoperation of an augmented reality device 104 for controlling a radio.The user 102 uses the AR device 104 to detect and identify a radio 1404,keys 1402, and a wallet 1406. The AR device 104 determines that the keys1402 and wallet 1406 are associated with the radio 1404 (for example,using unique visual or electronic identifiers on the keys 1402 andwallet 1406). The radio 1404 is tuned in to station A based on theplacement of the detected keys 1402 and wallet 1406. In the presentexample, the AR device 104 visually detects that the keys 1402, theradio 1404, and the wallet 1406 are located within a field of view 1202of an optical sensor of the AR device 104. The AR device 104 identifiesthat the position/placement of the keys 1402 and wallet 1406 relative tothe radio 1404. FIGS. 14A and 14B both illustrate the keys 1402 andwallet 1406 in different positions (e.g., to the left, to the right)relative to the radio 1404. The AR device 104 determines a command basedon the location of the keys 1402 and wallet 1406 relative to the radio1404 and sends the command to the radio 1404. In the example of FIG.14A, the AR device 104 sends a signal to the radio 1404 to tune tostation A based on detecting that the keys 1402 are placed to the leftof the radio 1404 and the wallet 1406 is placed to the right of theradio 1404. In the example of FIG. 14B, the AR device 104 sends a signalto the radio 1404 to tune to station B based on detecting that the keys1402 are placed to the right of the radio 1404 and the wallet 1406 isplaced to the left of the radio 1404.

FIG. 14C is a block diagram illustrating an example of an augmentedreality display for controlling a radio 1404. A display 1410 of the ARdevice 104 displays AR content overlaid on real world physical object108 (e.g., radio 1404). The AR content includes a radio stationidentification 1420 (e.g., logo of the station) that appears to be abovethe radio 1404. The AR content also includes illustrations or guide1440, 1450 on moving the keys 1402 and wallet 1406 to change the radio1404 to station B 1430.

Any one or more of the modules described herein may be implemented usinghardware (e.g., a processor of a machine) or a combination of hardwareand software. For example, any module described herein may configure aprocessor to perform the operations described herein for that module.Moreover, any two or more of these modules may be combined into a singlemodule, and the functions described herein for a single module may besubdivided among multiple modules. Furthermore, according to variousexample embodiments, modules described herein as being implementedwithin a single machine, database, or device may be distributed acrossmultiple machines, databases, or devices.

Modules, Components and Logic

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms. Modules may constitute eithersoftware modules (e.g., code embodied on a machine-readable medium or ina transmission signal) or hardware modules. A hardware module is atangible unit capable of performing certain operations and may beconfigured or arranged in a certain manner. In example embodiments, oneor more computer systems (e.g., a standalone, client, or server computersystem) or one or more hardware modules of a computer system (e.g., aprocessor or a group of processors) may be configured by software (e.g.,an application or application portion) as a hardware module thatoperates to perform certain operations as described herein.

In various embodiments, a hardware module may be implementedmechanically or electronically. For example, a hardware module maycomprise dedicated circuitry or logic that is permanently configured(e.g., as a special-purpose processor, such as a field programmable gatearray (FPGA) or an application-specific integrated circuit (ASIC)) toperform certain operations. A hardware module may also compriseprogrammable logic or circuitry (e.g., as encompassed within ageneral-purpose processor or other programmable processor) that istemporarily configured by software to perform certain operations. Itwill be appreciated that the decision to implement a hardware modulemechanically, in dedicated and permanently configured circuitry, or intemporarily configured circuitry (e.g., configured by software) may bedriven by cost and time considerations.

Accordingly, the term “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired) or temporarilyconfigured (e.g., programmed) to operate in a certain manner and/or toperform certain operations described herein. Considering embodiments inwhich hardware modules are temporarily configured (e.g., programmed),each of the hardware modules need not be configured or instantiated atany one instance in time. For example, where the hardware modulescomprise a general-purpose processor configured using software, thegeneral-purpose processor may be configured as respective differenthardware modules at different times. Software may accordingly configurea processor, for example, to constitute a particular hardware module atone instance of time and to constitute a different hardware module at adifferent instance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multipleof such hardware modules exist contemporaneously, communications may beachieved through signal transmission (e.g., over appropriate circuitsand buses that connect the hardware modules). In embodiments in whichmultiple hardware modules are configured or instantiated at differenttimes, communications between such hardware modules may be achieved, forexample, through the storage and retrieval of information in memorystructures to which the multiple hardware modules have access. Forexample, one hardware module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules may also initiate communications with input oroutput devices and can operate on a resource (e.g., a collection ofinformation).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions. The modulesreferred to herein may, in some example embodiments, compriseprocessor-implemented modules.

Similarly, the methods described herein may be at least partiallyprocessor-implemented. For example, at least some of the operations of amethod may be performed by one or more processors orprocessor-implemented modules. The performance of certain of theoperations may be distributed among the one or more processors, not onlyresiding within a single machine, but deployed across a number ofmachines. In some example embodiments, the processor or processors maybe located in a single location (e.g., within a home environment, anoffice environment or as a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

The one or more processors may also operate to support performance ofthe relevant operations in a “cloud computing” environment or as a“software as a service” (SaaS). For example, at least some of theoperations may be performed by a group of computers (as examples ofmachines including processors), these operations being accessible via anetwork and via one or more appropriate interfaces (e.g., APIs).

Electronic Apparatus and System

Example embodiments may be implemented in digital electronic circuitry,or in computer hardware, firmware, software, or in combinations of them.Example embodiments may be implemented using a computer program product,e.g., a computer program tangibly embodied in an information carrier,e.g., in a machine-readable medium for execution by, or to control theoperation of, data processing apparatus, e.g., a programmable processor,a computer, or multiple computers.

A computer program can be written in any form of programming language,including compiled or interpreted languages, and it can be deployed inany form, including as a stand-alone program or as a module, subroutine,or other unit suitable for use in a computing environment. A computerprogram can be deployed to be executed on one computer or on multiplecomputers at one site or distributed across multiple sites andinterconnected by a communication network.

In example embodiments, operations may be performed by one or moreprogrammable processors executing a computer program to performfunctions by operating on input data and generating output. Methodoperations can also be performed by, and apparatus of exampleembodiments may be implemented as, special purpose logic circuitry(e.g., a FPGA or an ASIC).

A computing system can include clients and servers 110. A client andserver 110 are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. Inembodiments deploying a programmable computing system, it will beappreciated that both hardware and software architectures meritconsideration. Specifically, it will be appreciated that the choice ofwhether to implement certain functionality in permanently configuredhardware (e.g., an ASIC), in temporarily configured hardware (e.g., acombination of software and a programmable processor), or a combinationof permanently and temporarily configured hardware may be a designchoice. Below are set out hardware (e.g., machine) and softwarearchitectures that may be deployed, in various example embodiments.

Example Machine Architecture and Machine-Readable Medium

FIG. 15 is a block diagram of a machine in the example form of acomputer system 1500 within which instructions for causing the machineto perform any one or more of the methodologies discussed herein may beexecuted. In alternative embodiments, the machine operates as astandalone device or may be connected (e.g., networked) to othermachines. In a networked deployment, the machine may operate in thecapacity of a server 110 or a client machine in a server-client networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment. The machine may be a personal computer (PC), atablet PC, a set-top box (STB), a personal digital assistant (PDA), acellular telephone, a web appliance, a network router, switch or bridge,or any machine capable of executing instructions (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute a set (or multiple sets) of instructions to perform anyone or more of the methodologies discussed herein.

The example computer system 1500 includes a processor 1502 (e.g., acentral processing unit (CPU), a graphics processing unit (GPU) orboth), a main memory 1504 and a static memory 1506, which communicatewith each other via a bus 1508. The computer system 1500 may furtherinclude a video display unit 1510 (e.g., a liquid crystal display (LCD)or a cathode ray tube (CRT)). The computer system 1500 also includes analphanumeric input device 1512 (e.g., a keyboard), a user interface (UI)navigation (or cursor control) device 1514 (e.g., a mouse), a disk driveunit 1516, a signal generation device 1518 (e.g., a speaker) and anetwork interface device 1520.

Machine-Readable Medium

The disk drive unit 1516 includes a machine-readable medium 1522 onwhich is stored one or more sets of data structures and instructions1524 (e.g., software) embodying or utilized by any one or more of themethodologies or functions described herein. The instructions 1524 mayalso reside, completely or at least partially, within the main memory1504 and/or within the processor 1502 during execution thereof by thecomputer system 1500, the main memory 1504 and the processor 1502 alsoconstituting machine-readable media 1522. The instructions 1524 may alsoreside, completely or at least partially, within the static memory 1506.

While the machine-readable medium 1522 is shown in an example embodimentto be a single medium, the term “machine-readable medium” may include asingle medium or multiple media (e.g., a centralized or distributeddatabase, and/or associated caches and servers 110) that store the oneor more instructions 1524 or data structures. The term “machine-readablemedium” shall also be taken to include any tangible medium that iscapable of storing, encoding or carrying instructions 1524 for executionby the machine and that cause the machine to perform any one or more ofthe methodologies of the present embodiments, or that is capable ofstoring, encoding or carrying data structures utilized by or associatedwith such instructions 1524. The term “machine-readable medium” shallaccordingly be taken to include, but not be limited to, solid-statememories, and optical and magnetic media. Specific examples ofmachine-readable media 1522 include non-volatile memory, including byway of example semiconductor memory devices (e.g., erasable programmableread-only memory (EPROM), electrically erasable programmable read-onlymemory (EEPROM), and flash memory devices); magnetic disks such asinternal hard disks and removable disks; magneto-optical disks; andcompact disc-read-only memory (CD-ROM) and digital versatile disc (ordigital video disc) read-only memory (DVD-ROM) disks.

Transmission Medium

The instructions 1524 may further be transmitted or received over acommunications network 1526 using a transmission medium. Theinstructions 1524 may be transmitted using the network interface device1520 and any one of a number of well-known transfer protocols (e.g.,HTTP). Examples of communication networks include a LAN, a WAN, theInternet, mobile telephone networks, POTS networks, and wireless datanetworks (e.g., Wi-Fi and WiMAX networks). The term “transmissionmedium” shall be taken to include any intangible medium capable ofstoring, encoding, or carrying instructions 1524 for execution by themachine, and includes digital or analog communications signals or otherintangible media to facilitate communication of such software.

Although an embodiment has been described with reference to specificexample embodiments, it will be evident that various modifications andchanges may be made to these embodiments without departing from thebroader spirit and scope of the present disclosure. Accordingly, thespecification and drawings are to be regarded in an illustrative ratherthan a restrictive sense. The accompanying drawings that form a parthereof, show by way of illustration, and not of limitation, specificembodiments in which the subject matter may be practiced. Theembodiments illustrated are described in sufficient detail to enablethose skilled in the art to practice the teachings disclosed herein.Other embodiments may be utilized and derived therefrom, such thatstructural and logical substitutions and changes may be made withoutdeparting from the scope of this disclosure. This Detailed Description,therefore, is not to be taken in a limiting sense, and the scope ofvarious embodiments is defined only by the appended claims, along withthe full range of equivalents to which such claims are entitled.

Such embodiments of the inventive subject matter may be referred toherein, individually and/or collectively, by the term “invention” merelyfor convenience and without intending to voluntarily limit the scope ofthis application to any single invention or inventive concept if morethan one is in fact disclosed. Thus, although specific embodiments havebeen illustrated and described herein, it should be appreciated that anyarrangement calculated to achieve the same purpose may be substitutedfor the specific embodiments shown. This disclosure is intended to coverany and all adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, will be apparent to those of skill in theart upon reviewing the above description.

The Abstract of the Disclosure is provided to comply with 37 C.F.R. §1.72(b), requiring an abstract that will allow the reader to quicklyascertain the nature of the technical disclosure. It is submitted withthe understanding that it will not be used to interpret or limit thescope or meaning of the claims. In addition, in the foregoing DetailedDescription, it can be seen that various features are grouped togetherin a single embodiment for the purpose of streamlining the disclosure.This method of disclosure is not to be interpreted as reflecting anintention that the claimed embodiments require more features than areexpressly recited in each claim. Rather, as the following claimsreflect, inventive subject matter lies in less than all features of asingle disclosed embodiment. Thus the following claims are herebyincorporated into the Detailed Description, with each claim standing onits own as a separate embodiment.

What is claimed is:
 1. An augmented-reality device comprising: adisplay; and one or more hardware processors configured to performoperations comprising: detecting that a physical device is locatedwithin a first threshold distance of the augmented-reality device;detecting that a physical object is located within a second thresholddistance of the physical device, the physical object beingelectronically unconnected to the physical device; and displaying, inthe display, an identification of a command to be performed by thephysical device based on a presence of the physical device and thephysical object, in response to detecting that the physical device islocated within the first threshold distance of the augmented-realitydevice, and that the physical object is located within the secondthreshold distance of the physical device.
 2. The augmented-realitydevice of claim 1, further comprising: an optical sensor, whereindisplaying the identification of the command further comprises:displaying virtual content as an overlay to the physical object, thevirtual content identifying the command.
 3. The augmented-reality deviceof claim 1, wherein the operations further comprise: determining thatthe physical object is associated with the physical device; andidentifying the command based on an identification of both the physicalobject and the physical device, the physical device configured tooperate the command.
 4. The augmented-reality device of claim 1, whereinthe operations further comprise: determining a position of the physicalobject relative to the physical device, a first orientation of thephysical object relative to the augmented-reality device, and a secondorientation of the physical object relative to the physical device,wherein the command is identified based on a combination of theidentification of the physical object, the position of the physicalobject relative to the physical device, the first orientation of thephysical object relative to the augmented-reality device, and the secondorientation of the physical object relative to the physical device. 5.The augmented-reality device of claim 1, wherein the operations furthercomprise: identifying the physical device and the physical object; anddetermining that the physical object is adjacent to the physical device,wherein the command is based on the identification of both the physicaldevice and the physical object, and the physical object being adjacentto the physical device.
 6. The augmented-reality device of claim 1,wherein the operations further comprise: wirelessly communicating withthe physical device; identifying the physical device based on thewireless communication between the augmented reality device and thephysical device; and determining that the physical object is adjacent tothe physical device, wherein the command is based on the identificationof both the physical device and the physical object, and the physicalobject being adjacent to the physical device.
 7. The augmented-realitydevice of claim 1, wherein the operations further comprise:communicating the command to the physical device using wirelesscommunication.
 8. The augmented-reality device of claim 2, wherein theoperations further comprise: detecting that the physical device hasperformed the command on the physical device; and modifying the virtualcontent in response to the detecting that the physical device hasperformed the command, the modified virtual content indicating that thephysical device has performed the command.
 9. The augmented-realitydevice of claim 2, wherein the operations further comprise: receiving aconfirmation from the physical device that the command has beenperformed on the physical device; and modifying the virtual content inresponse to the detecting that the physical device has performed thecommand, the modified virtual content indicating that the physicaldevice has performed the command.
 10. The augmented-reality device ofclaim 1, wherein the operations further comprise: detecting that a firstand a second physical object are located within the predefined distanceof the physical device, the first and second physical objects beingelectronically unconnected to the physical device; determining that thefirst and second physical objects are associated with the physicaldevice; identifying a first request based on a combination of anidentification of the first and second physical objects, the position ofthe first physical object relative to the physical device, the positionof the second physical object relative to the physical device, theorientation of the first physical object relative to theaugmented-reality device, and the orientation of the of the secondphysical object relative to the augmented-reality device; andcommunicating the first request to the physical device.
 11. A methodcomprising: detecting that a physical device is located within a firstthreshold distance of the augmented-reality device; detecting that aphysical object is located within a second threshold distance of thephysical device, the physical object being electronically unconnected tothe physical device; and displaying, in the display, an identificationof a command to be performed by the physical device based on a presenceof the physical device and the physical object, in response to detectingthat the physical device is located within the first threshold distanceof the augmented-reality device, and that the physical object is locatedwithin the second threshold distance of the physical device.
 12. Themethod of claim 11, wherein displaying the identification of the commandfurther comprises: displaying virtual content as an overlay to thephysical object, the virtual content identifying the command.
 13. Themethod of claim 11, further comprising: determining that the physicalobject is associated with the physical device; and identifying thecommand based on an identification of both the physical object and thephysical device, the physical device configured to operate the command.14. The method of claim 11, further comprising: determining a positionof the physical object relative to the physical device, a firstorientation of the physical object relative to the augmented-realitydevice, and a second orientation of the physical object relative to thephysical device, wherein the command is identified based on acombination of the identification of the physical object, the positionof the physical object relative to the physical device, the firstorientation of the physical object relative to the augmented-realitydevice, and the second orientation of the physical object relative tothe physical device.
 15. The method of claim 11, further comprising:identifying the physical device and the physical object; and determiningthat the physical object is adjacent to the physical device, wherein thecommand is based on the identification of both the physical device andthe physical object, and the physical object being adjacent to thephysical device.
 16. The method of claim 11, further comprising:wirelessly communicating with the physical device; identifying thephysical device based on the wireless communication between theaugmented reality device and the physical device; and determining thatthe physical object is adjacent to the physical device, wherein thecommand is based on the identification of both the physical device andthe physical object, and the physical object being adjacent to thephysical device.
 17. The method of claim 12, further comprising:detecting that the physical device has performed the command on thephysical device; and modifying the virtual content in response to thedetecting that the physical device has performed the command, themodified virtual content indicating that the physical device hasperformed the command.
 18. The method of claim 12, further comprising:receiving a confirmation from the physical device that the command hasbeen performed on the physical device; and modifying the virtual contentin response to the detecting that the physical device has performed thecommand, the modified virtual content indicating that the physicaldevice has performed the command.
 19. The method of claim 11, furthercomprising: detecting that a first and a second physical object arelocated within the predefined distance of the physical device, the firstand second physical objects being electronically unconnected to thephysical device; determining that the first and second physical objectsare associated with the physical device; identifying a first requestbased on a combination of an identification of the first and secondphysical objects, the position of the first physical object relative tothe physical device, the position of the second physical object relativeto the physical device, the orientation of the first physical objectrelative to the augmented-reality device, and the orientation of the ofthe second physical object relative to the augmented-reality device; andcommunicating the first request to the physical device.
 20. Anon-transitory machine-readable medium comprising instructions that,when executed by one or more processors of a machine, cause the machineto perform operations comprising: detecting that a physical device islocated within a first threshold distance of the augmented-realitydevice; detecting that a physical object is located within a secondthreshold distance of the physical device, the physical object beingelectronically unconnected to the physical device; and displaying, inthe display, an identification of a command to be performed by thephysical device based on a presence of the physical device and thephysical object, in response to detecting that the physical device islocated within the first threshold distance of the augmented-realitydevice, and that the physical object is located within the secondthreshold distance of the physical device.